Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training mini-batches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d.

In this paper, the authors propose Batch Renormalization technique to alleviate the problem of batchnorm when dealing with small or non-i.i.d minibatches. To reduce the dependence of large minibatch size is very important in many applications especially when training large neural network models with limited GPU memory. The proposed method is vey simple to understand and implement. And experiments show that Batch Renormalization performs well with non-i.i.d minibatches, and improves the results of small minibatches compared with batchnorm. Firstly, the authors give a clear review of batchnorm, and conclude that the key drawbacks of batchnorm are the inconsistency of mean and variance used in training and inference and the instability when dealing with small minibatches. Using moving averages to perform normalization would be the first thought, however this would lead to the model blowing up.

Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d.

The basic idea is to take a look at each activation after a layer and to normalize it by scaling and shifting it so that the mean and standard deviation across the current batch for that activation become 0 and 1, respectively. This is supposed to approximate a normalization with the population statistics by means of the batch statistics, leading to approximately normalized inputs for the following layer. That being said, a batch normalization layer is usually assumed to include a denormalization afterwards, that is, the normalized activations are once again transformed affinely so as to have a certain mean and standard deviation, which are learnable parameters of the model. This means that the inputs to the next layer are not normalized, but rather conform approximately to a mean and standard deviation that are independent of whatever the layer before the batch normalization layer produced. The benefits of batch normalization are manifest empirically, but their theoretical understanding is under debate. I will say no more about this as my intention is not to criticize the benefits, but to address the shortcomings of which there are also several: Memory consumption: All instances of the batch must be in memory at the same time in order to compute the batch statistics. This can become a problem if the data required per instance (the activations as well as the gradients of the activations with respect to loss) do not fit on the available hardware multiple times. Even if multiple devices are available, it requires either communication between these at each batch normalization layer, or to compromise on the accuracy of the batch statistics by computing it separately and independently for each device.

Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.